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Abstract 

The  US  Department  of  Defense  (DoD)  has  embraeed  the  concept  of  Joint  Experimentation  to 
help  identify  future  joint  requirements  and  potential  eapabilities  that  may  meet  those 
requirements.  Experimentation  is  viewed  as  a  means  to  spur  key  DoD  innovation,  to  help 
determine  DoD  priorities,  and  to  transition  potential  2U'  eentury  technology  and  process 
improvements  into  the  US  military. 

Despite  the  high  level  of  visibility  that  first  generation  Joint  and  Serviee-speeifie 
experimentation  programs  have  attracted,  surprisingly  little  attention  has  been  paid  to  how  to 
best  leverage  seientific  design  of  experiments  and  M&S  practiees  to  maximize  the  information 
that  ean  be  learned  from  sueh  experiments.  The  applieation  of  seientifie  methods  to  DoD 
experimentation  programs  is,  in  the  view  of  the  authors,  required  in  order  to  obtain  the  full 
benefit  possible  from  operational  experimentation. 

This  paper  presents  lessons  learned  from  the  applieation  of  the  Naval  Simulation  System  (NSS) 
and  general  design  of  experiment  praetiees  to  US  Navy  Eleet  Battle  Experiments  Alpha  through 
Echo.  NSS  has  been  involved  in  many  aspects  of  the  Navy  Experimentation  process  since  its 
ineeption.  These  applications  are  deseribed,  and  potential  future  applications  of  M&S  to  the 
Experiment  proeess  are  reeommended. 

1 .  Introduction 

In  the  past  fifteen  years,  the  US  Department  of  Defense  has  eome  to  reeognize  that  the  existing 
formal  aequisition  proeess  has  had  diffieulties  keeping  paee  with  improvements  in  teehnology. 
By  the  time  a  program  is  fielded,  the  state  of  the  art  -  and  indeed  the  requirement  for  the 
program  -  have  often  evolved  past  the  original  speeifications  for  the  system.  To  help  identify 
future  joint  requirements  and  potential  eapabilities  that  may  meet  those  requirements,  the  US 
Department  of  Defense  (DoD)  has  embraeed  the  eoneept  of  Joint  Experimentation. 
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Experimentation  is  viewed  as  a  uniquely  credible  and  robust  means  to  achieve  senior  Military 
and  Government  consensus  on  required  future  military  capabilities  and  the  defense  policies 
needed  to  realize  them.  This  credibility  stems  from  the  fact  that  experimentation  allows  current 
operational  forces  to  employ  new  technologies  and  new  organizational  command  structures  in  an 
operational  environment,  along  with  complementary  modeling  and  simulation  (M&S)  and  a 
scientific  approach,  to  quantify  the  value-added  of  specific  new  technologies  and  force  structures 
in  likely  operational  scenarios.  Hence,  Joint  Experimentation  is  viewed  as  a  means  to  spur  key 
DoD  innovation,  to  help  determine  DoD  priorities,  and  to  transition  potential  2E‘  century 
technology  and  process  improvements  into  the  US  military. 

Each  of  the  three  services  (including  the  Marine  Corps)  has  its  own  Experimentation  program. 
The  Elect  Battle  Experiment  (EBE)  series  was  initiated  when  the  Chief  of  Naval  Operations 
(CNO)  directed  the  Navy  to  engage  in  experiments  to  explore  new  weapons  systems, 
technologies,  and  employment  doctrines  to  carry  the  Navy  into  the  2U'  century.  Commander, 
Third  Elect  (C3E)  initiated  the  first  of  these,  EBE  Alpha,  in  March  of  1997  by  combining  a  naval 
surface  fire  support  (NSES)  power  projection  exercise  with  the  Commandant  of  the  Marine 
Corps  Advanced  Warfighting  Experiment  (AWE)  Hunter  Warrior.  Together,  these  experiments 
demonstrated  sea-based  command  and  control  of  a  special  Marine  Air-Ground  Task  Eorce 
(MAGTE)  and  examined  the  supporting  Command,  Control,  Computers,  and  Intelligence  (C4I) 
architecture  required  to  effectively  employ  NSES.  Operationally,  EBE  Alpha  examined  the 
feasibility  of  an  Arsenal  Ship  and  introduced  the  “Ring  of  Eire”  concept  for  large  scale 
coordinated  fire  control  services. 

In  EBEs  Bravo  and  Charlie,  conducted  in  September  1997  and  May  1998,  C3E  and  Commander, 
Second  Elect  (C2E)  respectively  have  continued  the  process  of  exploring  new  concepts  and 
technologies  by  further  investigating  the  Ring  of  Eire  construct  enhanced  with  Joint  tactical  air 
(TACAIR)  precision  strike  support  and  future  NSES  weapons.  EBE  Delta,  conducted  by 
Commander,  Seventh  Elect  (C7E)  in  October  1998,  examined  advanced  C2  concepts  and  systems 
relevant  to  the  counter- special  operations  forces  (CSOE)  and  Joint  Theater  Air  and  Missile 
Defense  (JTAMD)  mission  areas  for  the  first  time  in  a  forward  deployed  operational  theater. 
EBE  Echo,  recently  conducted  by  C3E,  examined  JTAMD  as  well  as  force  protection/port 
security  issues  against  an  asymmetric  (terrorist)  threat. 

Despite  the  high  level  of  visibility  that  first  generation  Joint  and  Service-specific 
experimentation  programs  have  attracted,  surprisingly  little  attention  has  been  paid  to  how  to 
best  leverage  scientific  design  of  experiments  and  M&S  practices  to  maximize  the  information 
that  can  be  learned  from  such  experiments.  The  application  of  scientific  methods  to  DoD 
experimentation  programs  is,  in  the  view  of  the  authors,  required  in  order  to  obtain  the  full 
benefit  possible  from  operational  experimentation. 

This  paper  presents  lessons  learned  from  the  application  of  the  Naval  Simulation  System  (NSS) 
and  general  design  of  experiment  practices  to  US  Navy  Elect  Battle  Experiments  Alpha  through 
Echo. 


2.  The  Naval  Simulation  System 


NSS  is  an  object  oriented,  Monte  Carlo  simulation  first  developed  in  the  early  90s  as  an  analytic 
tool  for  force  employment  and  course  of  action  analysis  for  Fleet  use.  It  models  entity-level 
interactions  with  a  specific  focus  on  C2,  i.e.  the  collection  of  information  by  sensors,  the 
dissemination  of  that  information  over  communications  networks,  the  fusion  of  that  information 
into  a  tactical  picture,  and  a  commander’s  decision  making  based  on  his  locally-held  tactical 
picture.  In  addition,  NSS  represents  detailed  command  structures  to  simulate  decision  making  at 
various  levels  within  a  command  architecture.  Commanders’  plans  are  represented  as  well  as 
individual  assets’  reactions  to  stimuli  within  their  perceived  environment. 

With  its  emphasis  on  C2,  NSS  was  first  tapped  to  support  the  FEE  series  in  the  role  of  scenario 
stimulator  for  C2  systems.  NSS  was  modified  to  generate  OTHT-Gold  contact  reports  which 
were  fed  to  C2  systems  being  evaluated  in  FBE-B.  A  single  replication  of  the  Monte  Carlo  NSS 
simulation  was  run  in  real-time  mode  and  the  tactical  picture  was  simulated  for  human  operators 
manning  the  C2  system. 

NSS  evolved  further  along  the  “scenario  generation/stimulation”  path  in  EBEs  Charlie  and  Delta, 
where  interfaces  to  the  model  were  developed  so  assets  simulated  in  NSS  could  be  manipulated 
in  real  time  in  response  to  tasking  from  the  human  operators  in  the  experiment.  In  EBE-D,  this 
interactive  mode  of  NSS  was  matured  to  the  point  where  the  NSS  operator  actually  “played” 
simulated  aircraft  by  making  voice  reports  over  real  communications  systems  to  live  operators 
tasking  them,  simulating  situation  reports,  battle  damage  assessments,  and  aircraft  checking  on 
station  for  tasking. 

Although  NSS  has  found  a  niche  in  the  EBE  process  in  scenario  stimulation,  it  also  has 
significant  capability  in  an  analysis  role.  This  capability  has  been  used  to  some  extent  in 
producing  quicklook  excursions,  performing  MOE  calculation,  and  in  post  exercise  analysis. 
However,  most  of  this  work  has  been  performed  on  an  ad-hoc  basis,  without  formal  tasking  from 
the  Maritime  Battle  Center  (the  agency  tasked  with  running  the  experiment  series)  and  the 
numbered  Elect  Commands  hosting  the  experiment. 

In  the  following  section,  we  will  discuss  the  specific  applications  of  NSS  to  various  roles  in  the 
Elect  Battle  Experiment  process.  We  divide  these  applications  into  three  phases  of  support:  pre¬ 
experiment,  experiment  execution,  and  post-experiment^  and  cite  specific  examples  of  using 
NSS  in  these  roles  in  the  five  EBEs  conducted  to  date.  References  [Stevens  and  Johnson,  1997], 
[Gagnon  and  Stevens,  1998(1)],  [Gagnon  and  Stevens,  1998(2)]  document  this  work. 

3.  NSS  Experience  in  the  FBE  Process  to  Date 

NSS  has  had  the  unique  experience  of  filling  various  roles  continuously  in  the  Elect  Battle 
Experiment  series,  from  its  inception.  We  cite  here  a  few  standout  functions  that  NSS  has 
performed  in  the  pre-experiment  phase,  the  experiment  itself,  and  in  post-experiment  analysis  as 
illustrative  examples  of  the  role  that  M&S  can  fill  in  operational  experimentation. 


'  This  closely  aligns  with  the  concept  of  “model-test-model”  being  considered  for  future  FBEs  and  for  use  in  the  Joint 
Experimentation  program. 


3 . 1  Pre-Experiment 


Prior  to  the  actual  execution  of  an  experiment,  several  things  have  to  happen.  The  technologies 
that  will  be  tested  must  be  identified  and  the  concepts  for  their  employment  must  be  thought  out. 
A  scenario  must  be  agreed  upon  which  is  both  of  interest  to  the  Fleet  Command  hosting  the 
experiment  and  which  will  be  a  good  context  for  the  proposed  tests.  Scenario  details  such  as 
Blue  and  Red  orders  of  battle,  time  frame  within  the  scenario,  number  and  duration  of 
experiment  runs  to  be  made,  and  data  collection  and  proposed  MOEs  for  analysis  must  be 
determined. 

NSS  has  been  employed  as  a  scenario  development  and  preview  tool  in  both  FBE  Charlie  and 
Echo,  using  a  different  approach  in  each  experiment.  In  EBE  Charlie,  where  Joint  Eires  were 
used  to  delay  an  enemy  advance  on  a  beach  head  being  secured  in  an  amphibious  assault, 
scenario  design  was  initially  performed  on  a  large  chart  where  enemy  columns’  tracks  were 
drawn  along  roads  and  across  bridges,  coordinated  with  movement  of  tactical  SAMs  and 
synchronized  with  Blue  transportation  of  troops  and  equipment  across  the  beach.  Once  the  data 
was  satisfactorily  drawn  out  on  paper,  it  was  entered  into  NSS  and  then  played  back  in  a  single 
replication  for  checking.  Errors  were  corrected,  tracks  were  adjusted,  and  timing  of  sorties  and 
movement  of  forces  were  modified  to  ensure  that  events  happened  in  the  desired  order.  The 
playback  capability  of  NSS  proved  to  be  an  extremely  useful  tool  for  review  of  the  scenario  and 
to  help  tailor  the  threat  presentation  for  the  desired  effect  for  the  experiment.  Figure  1  below 
shows  a  screen  snapshot  of  one  of  the  three  scenarios  NSS  represented  in  FBE  Charlie. 


Figure  1.  NSS  Scenario  Representation  for  FBE  Charlie 


In  FBE  Echo,  NSS  was  also  employed  prior  to  the  actual  experiment  to  explore  potential  tactics 
for  a  new  mission  area  for  which  tactics  really  did  not  exist.  Among  other  subjects,  FBE-E 
explored  a  port  security  scenario,  where  several  Blue  ships  were  anchored  in  a  port  and  defended 
by  patrol  boats  against  small-scale  attacks  by  a  non-standard  (“asymmetric”)  threat  such  as 


swimmers  laying  mines  or  terrorists  on  jet  skis  making  high  speed  attacks.  This  scenario  was 
entered  into  NSS  and  run  with  several  different  threat  presentations  to  explore  where  the 
proposed  defenses  broke  down.  The  idea  was  to  understand  the  problem  in  the  model  to  a  point 
where  (1)  a  challenging  and  realistic  threat  presentation  could  be  devised,  and  (2)  a  valid  defense 
strategy  could  be  suggested  for  testing  in  the  actual  experiment. 

FBE  Echo  was  the  first  time  that  M&S  was  used  to  explore  tactics  and  threats  in  this  manner 
prior  to  the  actual  experiment.  A  good  deal  more  could  have  been  done  with  the  scenario  in  NSS 
if  time  had  permitted.  Different  sensors,  command  and  control  arrangements,  and  even 
postulated  weapons  could  have  been  substituted  and  run  within  the  simulation.  Unfortunately, 
there  was  not  enough  time  to  perform  these  runs  prior  to  the  actual  experiment.  In  addition,  the 
insights  gathered  from  the  model  were  not  successfully  fed  into  the  experiment  planning  process 
due  to  insufficient  lead  time  and  the  absence  of  an  official  mechanism  for  this  transfer  of 
information.  However,  we  anticipate  that  this  approach  will  be  used  in  subsequent  experiments, 
starting  with  Eoxtrot  in  the  Eall  of  1999.  In  fact,  the  Naval  Postgraduate  School,  which  is 
performing  the  official  data  collection  and  analysis  for  the  EBE  series  starting  with  Echo,  has 
already  stated  a  desire  to  use  M&S  prior  to  the  experiment  to  mature  tactics  and  scenarios  “in  the 
laboratory”. 

3.2  Experiment  Execution 

Elect  Battle  Experiments  are  typically  held  concurrent  with  real-world  exercises  to  leverage  the 
assembly  of  live  forces.  Often  this  means  that  the  EBE  must  compromise  its  goals  because  it 
cannot  interfere  with  the  live  “host”  exercise.  One  way  for  the  EBEs  to  maximize  the 
opportunity  is  to  simulate  forces  that  are  not  available  from  the  host  exercise.  This  has  been  the 
main  focus  of  M&S  in  the  EBEs  to  date.  In  addition  to  “filling  in  the  gaps”  of  the  live  host 
exercise,  M&S  also  provides  a  mechanism  for  data  collection,  real-time  (quicklook)  analysis, 
and  training  of  operators  of  new  systems  under  evaluation.  NSS  has  been  used  in  EBEs  in  each 
of  these  roles.  We  will  now  discuss  these  uses  with  examples  from  different  experiments. 

In  Experiments  Bravo,  Charlie,  and  Delta,  NSS  teamed  with  the  Eand  Attack  Warfare  System 
(EAWS),  a  C2  system  that  allows  for  efficient  management  of  joint  fires  from  the  sea,  to 
examine  the  Ring  of  Eire  concept  for  joint  fires,  and  to  examine  the  use  of  precision  weapons 
against  time-critical  targets  in  JTAMD.  NSS  simulated  the  majority  of  the  scenario  in  each 
experiment,  typically  representing  the  following  assets  and  capabilities: 

■  Blue  ships,  subs,  forces  ashore,  and  aircraft; 

■  Red  order  of  battle,  including  surface,  air,  and  ground  assets; 

■  White  ships  and  aircraft  in  the  area; 

■  Green  (coalition)  ships,  aircraft,  and  forces  ashore; 

Blue  sensors  simulated  in  NSS  (including  JSTARS,  UAVs,  satellites,  Eorward  Air  Controllers, 
P-3s,  and  AW ACS)  sent  contact  reports  over  designated  communications  paths  to  nodes  were  the 
information  was  fused  into  a  tactical  picture.  A  simulated  command  node  declared  targets  from 
this  picture  “actionable”  for  attack  based  on  specific  criteria.  These  actionable  targets  were  then 


transmitted  to  the  LAWS  using  standard  OTH-Gold  reporting  formats  for  display  to  the 
operators  and  decision  makers.  Figure  2  shows  the  FEE  Bravo  C4ISR  architecture  for  the  Ring 
of  Fire  concept.  NSS  simulated  everything  to  the  left  of  the  dotted  line  between  the  Intel  and 
Fires  Cells.  The  arrangement  for  FEE  Charlie  was  very  similar. 


In  EBEs  Charlie  and  Delta,  EAWS  operators  tasked  (simulated)  airborne  Combat  Air  Support 
(CAS)  missions  to  attack  (simulated)  actionable  targets  on  the  battlefield.  When  the  tasking 
assignment  was  received,  NSS  “flew”  the  CAS  sorties  to  the  targets,  simulated  the  engagement, 
simulated  Battle  Damage  Assessments  (BDA)  by  sensors,  and  sent  the  EDA  reports  to  EAWS 

using  standard  message  formats  and  language.  NSS  also  reported  BDA  on  targets  engaged  by 

2 

cruise  missiles  and  guns,  weapons  represented  by  another  simulation  playing  in  the  experiment. 

In  EBEs  Bravo  and  Charlie,  NSS  essentially  executed  a  scripted  scenario.  Only  CAS  sorties  and 
some  target  motion  (in  Charlie)  were  dynamically  managed  from  the  simulation.  In  EBE  Delta, 
NSS’  execution  interface  was  greatly  expanded  to  allow  the  NSS  operator  to  respond  to  a  wide 
variety  of  requests  from  the  live  EAWS  operators  managing  a  maritime  offense  against  North 


^  The  Johns  Hopkins  Applied  Physics  Lah’s  APLSIM  was  a  high-fidelity  simulation  capable  of  representing  TLAM,  tactical 
TLAM,  gun  rounds,  and  other  flyout  weapons.  Detailed  weapons  trajectory  and  flyout  parameters  were  desired  so  airspace 
deconfliction  could  be  examined  in  the  Experiment.  NSS  and  APLSIM  were  linked  so  that  weapon  impact  events  in  APLSIM 
could  be  evaluated  in  NSS. 


Korean  seaborne  insertion  of  special  operations  forces.  Functions  NSS  simulated  dynamically  in 
FEE  Delta  included: 

■  Launch  of  alert  aircraft  specified  by  type,  launch  base,  and  callsign. 

■  Vectoring  of  attack  assets  against  targets;  attacks  could  be  focused  on  higher  priority 
targets  if  directed  by  LAWS  operators. 

■  Redirecting  of  sensors. 

■  Redirecting  of  ships  for  attack  or  threat  avoidance. 

The  realism  of  the  NSS  scenario  simulation  in  LEE  Delta  was  significantly  enhanced  by  the 
addition  of  voice  reports  accompanying  the  contact  reports  sent  to  LAWS.  A  voice  circuit 
between  the  NSS  and  LAWS  operators  allowed  the  following  reports  to  be  made: 

■  Situation  Reports  (SITREPS)  made  by  the  P-3  acting  as  airborne  controller  and  area 
sensor.  These  included  a  surface  picture  summary,  attack  assets  under  the  P-3’s 
control,  engagement  status  of  these  assets,  fuel  and  weapons  status,  and  any  unassigned 
contacts. 

■  Alert  aircraft  check-in  and  check-out  of  the  engagement  airspace. 

■  Eattle  damage  reports. 

The  details  of  the  NSS  simulation  and  the  accompanying  voice  reports  enhanced  the  realism  of 
the  presentation  to  the  LAWS  operators.  This  allowed  the  task  loading  of  the  decisionmakers 
using  the  LAWS  system  to  be  evaluated  realistically. 

When  NSS  is  used  to  drive  a  scenario,  it  is  in  a  unique  position  to  automatically  collect  data  that 
can  be  turned  around  for  a  quicklook  analysis  after  the  scenario  run  is  complete.  In  LEE  Eravo, 
data  collected  by  NSS  was  used  to  generate  MOEs  that  were  used  by  the  Maritime  Eattle  Center 
to  monitor  progress  and  to  determine  if  experiment  objectives  were  being  met.  NSS  automatic 
data  collection  and  MOE  generation  provided  details  on: 

■  Target  status  (targets  damaged  as  a  function  of  time  by  target  type,  and  the  distribution 
of  times  required  to  detect,  localize,  damage,  and  destroy  targets); 

■  Ring  of  Lire  Asset  utilization  (asset  assignment  versus  time  compared  to  the  number  of 
targets  within  range);  and 

■  Ring  of  Lire  C2  node  efficiency  (the  number  of  actionable  targets  held  by  LAWS 
compared  to  the  number  of  assets  allocated  by  LAWS,  over  the  course  of  a  run). 

Data  collection  for  final  report  generation  is  also  supported.  Lor  example,  in  LEE  Charlie,  NSS 
provided  data  in  electronic  format  to  the  Center  for  Naval  Analyses,  the  agency  tasked  with 
formal  data  collection  and  analysis  in  that  experiment. 

Another  very  important  function  filled  by  NSS  in  the  LEE  series  has  been  C2  system  operator 
training.  As  directed  by  the  CNO  charter,  the  LEEs  have  examined  new  C2  technologies  and 


innovative  C2  concepts,  which  are  typically  unfamiliar  to  the  operators  participating  in  the 
experiment.  Training  and  familiarization  with  the  equipment  and  procedures  has  been  necessary 
to  ensure  smoothness  of  operation  in  the  actual  experiment.  In  FBE  Charlie,  separate  training 
scenarios  were  developed  to  provide  a  simple  threat  presentation  for  operators  using  LAWS  for 
the  first  time.  In  FBE  Delta,  a  full  week  of  training  sessions  was  conducted  with  NSS  scenarios, 
involving  LAWS  operator  teams  with  various  experience  levels,  to  prepare  for  the  three  days  of 
actual  experiment  runs.  Without  these  training  sessions,  data  collected  during  the  actual 
experiment  would  be  tainted  by  operator  learning  curves. 

Another  unique  use  of  M&S  during  an  experiment  run  is  to  examine  excursions  to  the 
experiment  scenario  in  near-real-time.  This  is  often  impossible  because  of  the  short  turnaround 
time  and  the  requirement  to  enter  data  into  the  simulation  for  the  excursion.  However,  in  two 
separate  occasions,  NSS  analysts  were  able  to  use  the  simulation  to  generate  data  on  possible 
scenario  excursions  within  a  24-hour  period  during  the  experiment,  providing  insight  into  the 
data  being  collected.  In  FBE  Alpha,  NSS  was  run  to  examine  alternative  Arsenal  Ship  weapon 
loadouts  and  how  these  loadouts  would  change  the  preparation  of  the  battlefield  in  support  of 
tactical  air  strikes.  These  results  were  briefed  to  C3F  the  next  day.  In  FBE  Delta,  data  collected 
in  NSS  on  the  LAWS  operators’  times  to  make  decisions  for  allocations  of  assets  was  fed  back 
into  the  simulation.  Monte  Carlo  runs  were  then  made  with  excursions  on  both  attack  assets 
available  and  weapons  effectiveness  to  see  what  impact  these  parameters  would  have  on  the  rate 
at  which  targets  could  be  prosecuted.  Figure  3  shows  the  results  of  these  runs,  which  were 
provided  to  the  Maritime  Battle  Center  and  were  subsequently  cited  in  post-experiment  briefings 
and  reports.  The  data  with  diamonds  and  squares  marking  the  lines  are  “live”  experiment  data 
collected  by  NSS  during  scenario  runs.  The  data  without  diamonds  and  squares  were  generated 
by  Monte  Carlo  runs  of  NSS  with  the  excursions  noted  in  the  key.  NSS’s  simulated  LAWS 
operators’  performance  was  tuned  to  the  Team  B  performance  shown  in  the  plot. 
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Figure  3.  FBE  Delta  Excursion  Analysis  Performed  During  Experiment  Execution 


3.3  Post-Experiment 


As  a  source  of  data,  NSS  has  been  involved  in  post-experiment  analysis,  most  notably  in  FBE 
Bravo  and  Charlie  [Bravo,  Charlie  final  reports] .  However,  no  new  runs  of  the  model  were  made 
to  support  these  efforts.  In  FBE  Echo,  for  the  first  time,  the  NSS  scenario  developed  pre¬ 
experiment  is  being  used  to  examine  excursions  post-experiment  using  data  collected  during  the 
experiment.  Although  this  effort  is  ongoing  at  the  time  of  this  writing,  several  observations  can 
be  made  on  this  effort  and  its  utility  to  the  FBE  process. 

Data  collected  during  FBE  Echo  on  the  performance  of  new  sensors  tested  (target  detection  as  a 
function  of  range)  will  be  used  in  NSS  instead  of  estimates  of  sensor  performance  previously 
used.  Patrol  boat  defense  tactics  actually  employed  in  the  experiment  will  also  be  coded  into  the 
simulation  as  closely  as  possible  to  replicate  the  experiment  play.  Tracks  of  patrol  boats  and  the 
jet  ski  threats  were  recorded  using  global  positioning  system  (GPS)  units  during  experiment  runs. 
A  parser  routine  has  been  built  to  read  this  data  into  NSS  scenario  files  so  actual  tracks  can  also 
be  used  in  simulation  runs. 

One  way  NSS  has  immediately  shown  value  in  post-experiment  analysis  is  in  simply  replaying 
the  GPS-recorded  tracks.  By  simulating  the  proper  assets  on  the  tracks  and  running  the  model  in 
a  single  replication,  the  movement  of  assets  as  recorded  by  the  GPS  units  can  be  viewed.  This 
presentation,  combined  with  observer  notes  and  other  automated  data  collection,  is  essentially  a 
complete  historical  record  of  the  FBE  Echo  port  security  events.  As  the  saying  goes,  “A  picture 
is  worth  a  thousand  words”. 

Using  the  actual  FBE  tracks  and  sensor  performance  data,  the  FBE  events  can  be  simulated  in 
NSS  beyond  a  simple  playback.  By  allowing  the  model  to  simulate  communications,  data 
fusion,  and  the  centralized  commander’s  tactical  picture,  the  experiment  play  can  be  replicated 
and  expanded  upon  with  details  of  the  commander’s  perception  of  the  threat.  With  the 
simulation  set  up  in  this  way,  excursion  analysis  can  be  performed  by  simply  altering  aspects  of 
the  scenario.  With  the  FBE  Echo  analysis,  tentative  plans  are  to  run  different  weapons  systems, 
try  different  patrol  boat  patrol  regions,  vary  threat  presentations,  and  perhaps  explore  alternative 
command  arrangements.  In  this  way,  the  limited  set  of  data  collected  in  the  actual  experiment 
can  be  leveraged  into  a  wide  variety  of  lessons  learned  through  the  application  of  M&S.  In 
theory,  these  lessons  learned  can  be  re-tested  in  follow-on  experiments. 

4.  Recommendations:  What  M&S  Can  Still  Contribute  to  the  Experiment  Process 

Although  M&S  has  played  a  significant  role  in  each  stage  of  every  experiment  to  date,  there 
remain  some  important  areas  where  it  has  yet  to  improve  the  overall  process.  The  authors  are 
hopeful  that  as  the  Fleet  Battle  Experiment  series  matures,  M&S  will  be  applied  to  bring  more 
scientific  rigor  to  the  entire  process. 

Each  FBE  has  been  a  relatively  isolated  event  loosely  connected  to  previous  experiments  by  the 
common  theme  of  battle  management  using  future  C2  systems  such  as  EAWS.  FBE  Alpha 
through  Delta  increasingly  widened  the  scope  of  the  problem,  from  naval  surface  fire  support 
through  combined  fire  support  in  a  forward-deployed  theater.  Although  the  themes  have  been 


related,  there  has  been  no  effort  to  baseline  capabilities  and  experiment  off  that  baseline  with 
new  weapons,  sensors,  or  command  techniques.  This  is  the  essence  of  “experimentation”. 

The  concept  of  an  Experimental  Campaign,  which  is  receiving  serious  consideration  for  Joint 
Experimentation,  bears  some  discussion  here.  The  Campaign  links  a  series  of  experiments 
together,  with  each  subsequent  one  building  on  what  was  learned  from  previous  ones.  Topics  for 
future  experiments  are  derived  from  lessons  learned  in  past  experiments,  which  serve  as  a 
baseline  for  the  “new”  ideas.  This  rational  process  closely  mimics  the  scientific  premise  of 
establishing  a  known  and  understood  benchmark  and  then  measuring  deviations  from  that 
benchmark  to  identify  the  best  return  of  investment  with  respect  to  established  operational 
requirements  and  scenarios  of  interest. 

M&S  can  be  also  be  used  to  “screen”  likely  topics  for  future  experimentation.  Analytic  (Monte 
Carlo)  model  runs  considering  new  proposed  operational  concepts  can  be  made  using  a  familiar 
baseline  scenario.  Results  would  provide  a  measure  of  the  likely  improvements  obtained  using 
the  proposed  advanced  concept.  Given  that  significant  improvements  are  possible,  one  could 
then  identify  existing  prototype  C2  systems  consistent  with  the  new  operational  concept.  These 
systems  could  then  be  investigated  further  through  wargaming  or  similar  techniques.  The  most 
promising  systems  would  then  be  nominated  for  subsequent  operational  experimentation. 

Eor  example,  referring  back  to  the  EBE  Delta  counter- special  operations  forces  (SOE)  scenario, 
suppose  that  M&S  analytic  runs  indicated  that  significant  speed  of  command  improvements 
could  be  made  if  the  centralized  maritime  commander  responsible  for  prosecuting  the  SOE 
forces  could  be  decentralized  into  multiple  command  functions.  The  next  step  would  be  to 
identify  potential  C2  systems  that  would  enable  distributed  command  and  control  in  that  mission 
area.  These  systems  could  be  linked  together  in  the  laboratory  with  M&S  systems  capable  of 
providing  scenario  stimulation  and  of  simulating  resulting  tasking  orders  to  Blue  assets.  Metrics 
evaluating  likely  warfighting  performance  improvements  with  respect  to  baseline  performance 
could  be  calculated,  and  the  most  promising  C2  systems  could  be  identified  for  future  “live” 
experimentation . 

Post-experiment,  M&S  could  be  used  to  stress-test  concepts  for  larger-scale  scenarios.  The 
result  would  be  to  show  there  is  reason  to  believe  that  the  new  concepts  could  meet  identified 
operational  requirements. 

Employing  a  consistent  M&S  framework  throughout  the  experiment  process  -  from  pre¬ 
experiment  modeling,  through  at-sea  experimentation,  to  subsequent  post-experiment  analysis  - 
allows  for  consistency  in  evaluation  both  within  the  single  experiment  and  throughout  the 
Experimental  Campaign.  The  Defense  Modeling  and  Simulation  Office  (DMSO)  has  developed 
the  High  Eevel  Architecture  (HEA)  standards,  practices,  and  federation  software  for  just  such  a 
purpose.  Multiple  C4ISR  simulations,  C2  systems,  display  systems,  and  post-analysis  systems 
could  interact  using  a  standard,  documented  interface,  which  would  enhance  analysis,  wargame, 
and  experiment  credibility  and  repeatability.  A  HEA  M&S  framework  should  receive  serious 
consideration  for  future  experimentation  efforts. 


^  Indeed,  the  NSS  IT-21  Warfighting  Assessment  study  conducted  for  CINCPACFLT  in  November  1998  determined  that 
distributed  command  would  result  in  major  timeline  improvements  in  the  counter-SOF  mission  for  that  theater  [Atamian,  1999]. 


5.  Summary 


Modeling  and  simulation  has  played  an  important  role  in  the  Navy’s  Fleet  Battle  Experiment 
program  since  its  inception.  M&S  has  been  active  in  pre-experiment  analysis,  has  been 
instrumental  in  experiment  execution,  and  has  contributed  to  post-experiment  analysis  and 
interpretation  of  results. 

The  authors  believe  that  M&S  should  play  an  even  greater  role  by  providing  a  comprehensive 
environment  for  Joint  and  Navy  experimentation.  This  environment  can  support  advanced  C2 
concept  generation,  assessment,  wargaming,  and  operational  experimentation,  including 
seamless  transition  from  one  activity  to  the  next,  by  providing  a  consistent  framework.  The 
software  required  to  realize  this  vision  already  exists  to  a  large  extent.  The  DMSO  HLA 
construct  for  simulation  and  C2  system  interoperability  has  been  demonstrated  to  provide  an 
effective  interface  for  linking  systems  in  a  manner  which  well  suits  the  experimentation  process. 
The  Naval  Simulation  System,  which  has  already  established  itself  in  the  FBE  process  as  a  key 
tool  for  experimentation,  is  one  of  many  HEA-compliant  simulations  that  could  be  integrated  to 
support  experimentation.  Developing  a  software  framework  and  an  experimental  concept  for 
cradle-to-grave  concept  testing  will  result  in  a  more  robust  test  process  and  add  credibility  to  any 
experimental  program. 
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